{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "CS194-16 Introduction to Data Science\n",
      "\n",
      "**Name**: *Please put your name*\n",
      "\n",
      "**Student ID**: *Please put your student ID*\n",
      "\n",
      "\n",
      "Assignment 1: Text Analysis and Entity Resolution\n",
      "===\n",
      "\n",
      "## Overview\n",
      "\n",
      "Entity resolution is a common, yet difficult problem in data cleaning and integration.\n",
      "In this assignment, we will use powerful and scalable text analysis techniques to perform entity resolution across two data sets of commercial products."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Entity Resolution\n",
      "\n",
      "Entity resolution, also known as record deduplication, is the process of identifying rows in one or more data sets that refer to the same real world entity.\n",
      "Take an example.\n",
      "You're on ebay looking for a [hip data science accessory][sliderulewiki], but you're on a budget, so you decide to scrape the [ebay listings][ebay] for a few days to get a feel for the market.\n",
      "Unfortunately, the listings are confusing and you don't know how to aggregate them.\n",
      "Entity resolution to the rescue!\n",
      "You find an [authoritative database][sliderules] and map all the ebay listings to it.\n",
      "Now you can comparison shop, get that sweet Keuffel and Esser for way cheap, and impress all the data hipsters.\n",
      "\n",
      "But finding matching records is a hard problem in general.\n",
      "A major reason is that the criteria for identifying duplicates are often vague and impossible to encode in rules.\n",
      "In addition, from a purely computational perspective, the problem is quadratic in the size of its inputs: naively, all pairs of records need to be compared to find all the duplicates.\n",
      "In this assignment, we will begin to address both these challenges.\n",
      "\n",
      "[sliderulewiki]: http://en.wikipedia.org/wiki/Slide_rule \"Only 2X faster than Hadoop\"\n",
      "\n",
      "[sliderules]: http://www.sliderule.ca/ \"Artisanal Data Technology\"\n",
      "\n",
      "[ebay]: http://www.ebay.com/sch/i.html?_odkw=keuffel+esser+slide+rule&_osacat=0&_trksid=p2045573.m570.l1313.TR0.TRC0.Xslide+rule&_nkw=slide+rule&_sacat=0&_from=R40 \"Impress your colleagues\""
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "### Application\n",
      "\n",
      "Your assignment is to perform entity resolution over two web-scraped data sets of commercial product listings, one from Amazon, and one from Google.\n",
      "The goal is to build a unified database of all products listed on the Internet: a one-stop-shop for all your shopping needs.  (Elevator pitch: *it's like **Kayak.com** for **e-commerce**!*)\n",
      "\n",
      "The web has made unprecedented amounts of data available publicly, but scraped data frequently needs to be de-duplicated.\n",
      "These data sets are typical examples of what you can collect with some simple scripting.\n",
      "The data is not especially large (just a few thousand records), but even so, you will find that entity resolution is a major challenge (top results with this data are ~50% success rate).\n",
      "Don't get discouraged; the goal is to get acquainted with techniques to tackle the problem, and apply them to a representative example.\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Files\n",
      "\n",
      "Data files for this assignment can be found at:\n",
      "\n",
      "`https://github.com/amplab/datascience-sp14/raw/master/hw1/hw1data.tar.gz`\n",
      "\n",
      "The zip file includes the following files:\n",
      "\n",
      "* **Google.csv**, the Google Products data set \n",
      "* **Amazon.csv**, the Amazon data set\n",
      "* **Google_small.csv**, 200 records sampled from the Google data\n",
      "* **Amazon_small.csv**, 200 records sampled from the Amazon data\n",
      "* **Amazon_Google_perfectMapping.csv**, the \"gold standard\" mapping\n",
      "* **stopwords.txt**, a list of common English words\n",
      "\n",
      "Besides the complete data files, there are \"sample\" data files for each data set.\n",
      "Use these for **Part 1**.\n",
      "In addition, there is a \"gold standard\" file that contains all of the true mappings between entities in the two data sets.\n",
      "Every row in the gold standard file has a pair of record IDs (one Google, one Amazon) that belong to two record that describe the same thing in the real world.\n",
      "We will use the gold standard to evaluate our algorithms."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Deliverables\n",
      "\n",
      "Complete the all the exercises below and turn in a write up in the form of an IPython notebook, that is, **an .ipynb file**.\n",
      "The write up should include your code, answers to exercise questions, and plots of results.\n",
      "Complete submission instructions will be posted on Piazza.\n",
      "\n",
      "You can use this notebook and fill in answers inline, or if you prefer, do your write up in a separate notebook.\n",
      "In this notebook, we provide code templates for many of the exercises.\n",
      "They are intended to help with code re-use, since the exercises build on each other, and are highly recommended.\n",
      "Don't forget to include answers to questions that ask for natural language responses, i.e., in English, not code!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Guidelines\n",
      "\n",
      "#### Code\n",
      "\n",
      "This assignment can be done with basic python and matplotlib.\n",
      "Feel free to use PANDAs, too, which you may find well suited to several exercises.\n",
      "As for other libraries, please check with course staff whether they're allowed.\n",
      "In general, we want you to use whatever is comfortable, except for libraries (e.g., NLTK) that include functionality covered in the assignment.\n",
      "\n",
      "You're not required to do your coding in IPython, so feel free to use your favorite editor or IDE.\n",
      "But when you're done, remember to put your code into a notebook for your write up.\n",
      "\n",
      "#### Collaboration\n",
      "\n",
      "This assignment is to be done individually.  Everyone should be getting a hands on experience in this course.  You are free to discuss course material with fellow students, and we encourage you to use Internet resources to aid your understanding, but the work you turn in, including all code and answers, must be your own work."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Part 0: Preliminaries\n",
      "\n",
      "### Exercise 0\n",
      "\n",
      "Download the data and unzip it.\n",
      "Read each file in from the file system, and store them as lists of lines.\n",
      "\n",
      "For each of the data files (\"Google.csv\", \"Amazon.csv\", and the samples), we want to parse the IDs out of each record.\n",
      "The IDs are the first column of the file (they are URLs for Google, and alphanumeric strings for Amazon).\n",
      "Omitting the headers, load these data files into *dictionaries mapping ID to a string containing the rest of the record*.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "DATA_PATH = \"\" # Make this the /path/to/the/data\n",
      "\n",
      "# TODO Load data files here..."
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Part 1: ER as Text Similarity\n",
      "\n",
      "A simple approach to entity resolution is to treat all records as strings and compute their similarity with a string distance function.\n",
      "In this section, we will build some components for bag-of-words text-analysis, and use them to compute record similarity.\n",
      "\n",
      "### 1.1 Bags of Words\n",
      "\n",
      "Bag-of-words is a conceptually simple yet powerful approach to text analysis.\n",
      "The idea is to treat strings, a.k.a. **documents**, as *unordered collections* of words, or **tokens**, i.e., as bags of words.\n",
      "\n",
      "> **Note on terminology**: \"token\" is more general than what we ordinarily mean by \"word\" and includes things like numbers, acronyms, and other exotica like word-roots and fixed-length character strings.\n",
      "> Bag of words techniques all apply to any sort of token, so when we say \"bag-of-words\" we really mean \"bag-of-tokens,\" strictly speaking.\n",
      "\n",
      "Tokens become the atomic unit of text comparison.\n",
      "If we want to compare two documents, we count how many tokens they share in common.\n",
      "If we want to search for documents with keyword queries (this is what Google does), then we turn the keywords into tokens and find documents that contain them.\n",
      "\n",
      "The power of this approach is that it makes string comparisons insensitive to small differences that probably do not affect meaning much, for example, punctuation and word order.\n",
      "\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Exercise 1\n",
      "\n",
      "**a**. Implement the function `simple_tokenize(string)` that takes a string and returns a list of tokens in the string.\n",
      "`simple_tokenize` should split strings using the provided regular expression.\n",
      "Since we want to make token-matching case insensitive, make sure all tokens are lower-case.\n",
      "Give an interpretation, in natural language, of what the regular expression, `split_regex`, matches."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import re\n",
      "\n",
      "quickbrownfox = \"A quick brown fox jumps over the lazy dog.\"\n",
      "split_regex = r'\\W+'\n",
      "\n",
      "# TODO Implement this\n",
      "def simple_tokenize(string):\n",
      "    pass\n",
      "\n",
      "print simple_tokenize(quickbrownfox) # Should give ['a', 'quick', 'brown', ... ]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b**. *Stopwords* are common words that do not contribute much to the content or meaning of a document (e.g., \"the\", \"a\", \"is\", \"to\", etc.).\n",
      "Stopwords add noise to bag-of-words comparisons, so the are usually excluded.\n",
      "Using the included file \"stopwords.txt\", implement `tokenize`, an improved tokenizer that does not emit stopwords.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "stopwords = [] # Load from file\n",
      "\n",
      "# TODO Implement this\n",
      "def tokenize(string):\n",
      "    pass\n",
      "\n",
      "print tokenize(quickbrownfox) # Should give ['quick', 'brown', ... ]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c**. Now let's tokenize the two small data sets.\n",
      "For each one build a dictionary of tokens, i.e., a dictionary where the record IDs are the keys, and the output of `tokenize` is the values.\n",
      "How many tokens, total, are there in the two data sets?\n",
      "Which Amazon record has the biggest number of tokens?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Compute these (dict() or DataFrame OK)\n",
      "amazon_rec2tok = {}\n",
      "google_rec2tok = {}\n",
      "\n",
      "total_tokens = 0 # TODO Fix me\n",
      "print 'There are %s tokens in the combined data sets' % total_tokens\n",
      "\n",
      "biggest_record = \"\" # TODO Fix me\n",
      "print 'The Amazon record with ID \"%s\" has the most tokens' % biggest_record"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### 1.2 Weighted Bag-of-Words: TF-IDF\n",
      "\n",
      "Bag-of-words comparisons are not very good when all tokens are treated the same: some tokens are more important than others.\n",
      "Weights give us a way to specify which tokens to favor.\n",
      "With weights, when we compare documents, instead of counting common tokens, we sum up the weights of common tokens.\n",
      "\n",
      "A good heuristic for assigning weights is called \"Term-Frequency/Inverse-Document-Frequency,\" or TF-IDF for short.\n",
      "\n",
      "#### TF\n",
      "\n",
      "TF rewards tokens that appear many times in the same document.\n",
      "It is computed as the frequency of a token in a document, that is, if document `d` contains 100 tokens and token `t` appears in `d` 5 times, then the TF weight of `t` in `d` is `5/100 = 1/20`.\n",
      "The intuition for TF is that if a word occurs often in a document, then it is more important to the meaning of the document.\n",
      "\n",
      "#### IDF\n",
      "\n",
      "IDF rewards tokens that are rare overall in a data set.\n",
      "The intuition is that it is more significant if two documents share a rare word than a common one.\n",
      "IDF weight for a token, *t*, in a set of documents, *U*, is computed as follows: \n",
      "\n",
      "* Let *N* be the total number of documents in *U*\n",
      "* Find *n(t)*, the number of documents in *U* that contain *t*\n",
      "* Then *IDF(t) = N/n(t)*.\n",
      "\n",
      "Note that *n(t)/N* is the frequency of *t* in *U*, and *N/n* is the inverse frequency.\n",
      "\n",
      "> **Note on terminology**: Sometimes token weights depend on the document the token belongs to, that is, the same token may have a different weight when it's found in different documents.  We call these weights `local` weights.  TF is an example of a local weight, because it depends on the length of the source.  On the other hand, some token weights only depend on the token, and are the same everywhere that token is found.  We call these weights `global`, and IDF is one such weight.\n",
      "\n",
      "#### TF-IDF\n",
      "\n",
      "Finally, to bring it all together, the total TF-IDF weight for a token in a document is the product of its TF and IDF weights."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Exercise 2\n",
      "\n",
      "**a**. Implement `tf(tokens)` that takes a list of tokens belonging to a single document and returns a dictionary mapping tokens to TF weights.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this\n",
      "def tf(tokens):\n",
      "    pass\n",
      "\n",
      "print tf(tokenize(quickbrownfox)) # Should give { 'quick': 0.1666 ... }"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b**. Implement `find_idfs` that assigns an IDF weight to every unique token in a collection of data called `corpus`.  You may structure `corpus` however you want, but `find_idfs` should return a dictionary mapping tokens to weights.  Use `find_idfs` to compute IDF weights for all tokens in the combined small data sets.  How many unique tokens are there?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this\n",
      "def find_idfs(corpus):\n",
      "    pass\n",
      "\n",
      "idfs_small = {} # Use find_idfs here\n",
      "\n",
      "unique_tokens = 0 # Fix me\n",
      "\n",
      "print \"There are %s unique tokens in the small data sets.\" % unique_tokens\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c**. What are the 10 tokens with the smallest IDF in the combined small data set?  Do you think they are useful for entity resolution?  Why or why not?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "small_idf_tokens = [] # TODO Compute me\n",
      "\n",
      "print small_idf_tokens"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**d**. Plot a histogram of IDF values.  Be sure to use appropriate scaling and bucketing for the data.  What conclusions can you draw from the distribution of weights?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pylab\n",
      "%matplotlib inline\n",
      "\n",
      "# TODO Make a plot. HINT: You can use pylab.hist"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**e**. Use `tf` to implement `tfidf(tokens, idfs)` that takes a list of tokens from a document and a dictionary of idf weights and returns a dictionary mapping tokens to total TF-IDF weight.  Use `tfidf` to compute the weights of Amazon product record 'b000hkgj8k'."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this\n",
      "def tfidf(tokens, idfs):\n",
      "    pass\n",
      "\n",
      "rec_b000hkgj8k_weights = tfidf(None, None) # Fix me\n",
      "\n",
      "print \"Amazon record 'b000hkgj8k' has tokens and weights:\\n%s\" % rec_b000hkgj8k_weights"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "### 1.3 Cosine Similarity\n",
      "\n",
      "Now we are ready to do text comparisons in a formal way.\n",
      "The metric of string distance we will use is called **cosine similarity**.\n",
      "We will treat each document as a vector in some high dimensional space.\n",
      "Then, to compare two documents we compute the cosine of the angle between their two document vectors.\n",
      "This is easier than it sounds.\n",
      "\n",
      "The first question to answer is how do we represent documents as vectors?\n",
      "The answer is familiar: bag-of-words!\n",
      "We treat each unique token as a dimension, and treat token weights as magnitudes in their respective token dimensions.\n",
      "For example, suppose we use simple counts as weights, and we want to interpret the string \"Hello, world!  Goodbye, world!\" as a vector.\n",
      "Then in the \"hello\" and \"goodbye\" dimensions the vector has value 1, in the \"world\" dimension it has value 2, and it is zero in all other dimensions.\n",
      "\n",
      "Next question is: given two vectors how do we find the cosine of the angle between them?\n",
      "Recall the formula for the dot product of two vectors:\n",
      "\n",
      "$$a \\cdot b = \\| a \\| \\| b \\| \\cos \\theta$$\n",
      "\n",
      "Here $a \\cdot b = \\sum_{i=1}^n a_i b_i$ is the ordinary dot product of two vectors, and $\\|a\\| = \\sqrt{ \\sum_{i=1}^n a_i^2 }$ is the norm of $a$.\n",
      "\n",
      "We can rearrange terms and solve for the cosine to find it is simply the normalized dot product of the vectors.\n",
      "With our vector model, the dot product and norm computations are simple functions of the bag-of-words document representations, so we now have a formal way to compute similarity:\n",
      "\n",
      "$$similarity = \\cos \\theta = \\frac{a \\cdot b}{\\|a\\| \\|b\\|} = \\frac{\\sum_{i=1}^n a_i b_i}{\\sqrt{\\sum_{i=1}^n a_i^2} \\sqrt{\\sum_{i=1}^n b_i^2}}$$\n",
      "\n",
      "Setting aside the algebra, the geometric interpretation is more intuitive.\n",
      "The angle between two document vectors is small if they share many tokens in common, because they are pointing in roughly the same direction.\n",
      "Then, the cosine of the angle will be large.\n",
      "Otherwise, if the angle is large (and they have few words in common), the cosine is small.\n",
      "So the cosine scales proportionally with our intuitive sense of similarity."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "#### Exercise 3\n",
      "\n",
      "**a**. Implement `cosine_similarity(string1, string2, idfs)` that takes two strings and computes their cosine similarity in the context of some global IDF weights.\n",
      "Use `tokenize`, `tfidf`, and the IDF weights from exercise **2b** for extracting tokens and assigning them weights.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import math\n",
      "\n",
      "# Optional utility\n",
      "def dotprod(a, b):\n",
      "    pass\n",
      "\n",
      "# Optional utility\n",
      "def norm(a):\n",
      "    pass\n",
      "\n",
      "# Optional freebie\n",
      "def cossim(a, b):\n",
      "    return dotprod(a, b) / norm(a) / norm(b)\n",
      "\n",
      "test_vec1 = {'foo': 2, 'bar': 3, 'baz': 5 }\n",
      "test_vec2 = {'foo': 1, 'bar': 0, 'baz': 20 }\n",
      "print dotprod(test_vec1, test_vec2), norm(test_vec1) # Should be 102 6.16441400297\n",
      "\n",
      "# TODO Implement this\n",
      "def cosine_similarity(string1, string2, idfs):\n",
      "    pass\n",
      "\n",
      "print cosine_similarity(\"Adobe Photoshop\",\n",
      "                        \"Adobe Illustrator\", \n",
      "                        idfs_small) # Should be 0.0577243382163"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b**. Now we can finally do some entity resolution!\n",
      "For every product record in the small Google data set, use `cosine_similarity` to compute its similarity to every record in the small Amazon set.  Build a dictionary mapping `(Amazon Id, Google Id)` tuples to similarity scores between 0 and 1.\n",
      "What is the similarity between Amazon record 'b000o24l3q' and Google record `http://www.google.com/base/feeds/snippets/17242822440574356561`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Compute similarities\n",
      "similarities = {}\n",
      "\n",
      "print 'Requested similarity is %s.' % similarities[('b000o24l3q',\n",
      "  'http://www.google.com/base/feeds/snippets/17242822440574356561')]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c**.  Use the \"gold standard\" data (loaded from the included file) to answer the following questions.  How many true duplicate pairs are there in the small data set?  What is the average similarity score for true duplicates?  What about for non-duplicates?  Based on this, is cosine similarity doing a good job, qualitatively speaking, of identifying duplicates?  Why or why not?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gold_standard = [] # Load this if not already loaded\n",
      "\n",
      "true_dups = 0 # Fix me\n",
      "avg_sim_dups = 0.0 # Fix me\n",
      "avg_sim_non = 0.0 # Fix me\n",
      "\n",
      "print \"There are %s true duplicates.\" % true_dups\n",
      "print \"The average similarity of true duplicates is %s.\" % avg_sim_dups\n",
      "print \"And for non duplicates, it is %s.\" % avg_sim_non\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Part 2: Scalable ER\n",
      "\n",
      "In the previous section we built a text similarity function and used it for small scale entity resolution.  Our implementation is limited by its quadratic run time complexity, and is not practical for even modestly sized data sets.  In this section we will implement a more scalable algorithm and use it to do entity resolution on the full data set.\n",
      "\n",
      "### Inverted Indices\n",
      "\n",
      "To improve our ER algorithm from **Part 1**, we should begin by analyzing its running time.\n",
      "In particular, the algorithm above is quadratic in two ways.\n",
      "First, we did a lot of redundant computation of tokens and weights, since each record was reprocessed every time it was compared.\n",
      "Second, we made qudratically many token comparisons between records.\n",
      "\n",
      "The first source of quadratic overhead can be eliminated with precomputation and look-up tables, but the second source is a little more tricky.\n",
      "In the worst case, every token in every record in one data set exists in every record in the other data set, and therefore every token makes a nonzero contribution to the cosine similarity.\n",
      "In this case, token comparison is unavoidably quadratic.\n",
      "\n",
      "But in reality most records have nothing (or very little) in common.\n",
      "Moreover, it is typical for a record in one data set to have at most one duplicate record in the other data set (this is the case assuming each data set has been de-duplicated against itself).\n",
      "In this case, the output is linear in the size of the input and we can hope to achieve linear running time.\n",
      "\n",
      "An **inverted index** is a data structure that will allow us to avoid making quadratically many token comparisons.  It maps each token in the data set to the list of documents that contain the token.  So, instead of comparing, record by record, each token to every other token to see if they match, we will use inverted indices to *look up* records that match on a particular token.\n",
      "\n",
      "> **Note on terminology**: In text search, a *forward* index maps documents in a data set to the tokens they contain.  An *inverted* index supports the inverse mapping.\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Exercise 4\n",
      "\n",
      "> **Note**: For this section, use the complete Google and Amazon data sets, not the samples\n",
      "\n",
      "> **Pandas note**: If you use DataFrames for the mapping tables, make sure you index them correctly for efficient key look ups\n",
      "\n",
      "**a**. To address the overhead of recomputing tokens and token-weights, build a dictionary for each data set that maps record IDs to TF-IDF weighted token vectors (the vectors themselves should be dictionaries).  You will need to re-use code from above to recompute IDF weights for the complete combined data set."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Redo tokenization for full data set\n",
      "amazon_rec2tok = {}\n",
      "google_rec2tok = {}\n",
      "\n",
      "# TODO Recompute IDFs for full data set\n",
      "idfs = {}\n",
      "\n",
      "# TODO Pre-compute TF-IDF weights.  Build mappings from record ID weight vector.\n",
      "google_weights = {}\n",
      "amazon_weights = {}\n",
      "\n",
      "# TODO Pre-compute norms.  Build mappings from record ID to norm of the weight vector.\n",
      "google_norms = {}\n",
      "amazon_norms = {}\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b**. Build inverted indices of both data sources."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this. Should return a mapping from token to list-of-record-IDs\n",
      "def invert_index(forward_index):\n",
      "    pass\n",
      "\n",
      "# TODO Pre-compute inverted indices\n",
      "amazon_inv = invert_index(amazon_weights)\n",
      "google_inv = invert_index(google_weights)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c**. We are now in position to efficiently perform ER on the full data sets.\n",
      "Implement the following algorithm to build a dictionary that maps a pair of records (as a tuple) to a list of tokens they share in common:\n",
      "Iterate over tokens of one data set, and for each token, if the token appears in the other data set, use the inverted indices to find all pairs of records (one from either set) that contain the token.  Add these pairs to the output."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import itertools\n",
      "\n",
      "# TODO Implement algorithm to compute this:\n",
      "common_tokens = {} # Should map a record ID pair to a list of common tokens\n",
      "\n",
      "print len(common_tokens) # Should be 2882143"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**d**. Use the data structures from parts **a** and **c** to build a dictionary to map record pairs to cosine similarity scores."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this. Should take two record IDs and a list of common\n",
      "# tokens and return the cosine similarity of the two records.\n",
      "# Use results from part *a* for fast look ups.\n",
      "def fast_cosine_similarity(a_rec, g_rec, tokens):\n",
      "    pass\n",
      "\n",
      "# TODO Compute similarities (use fast_cosine_similarity)\n",
      "sims = {} # Should map record-ID-pairs to cosine similarity score\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "### Analysis\n",
      "\n",
      "Now we have an authoritative list of record-pair similarities, but we need a way to use those similarities to decide if two records are duplicates or not.\n",
      "The simplest approach is to pick a **threshold**.\n",
      "Pairs whose similarity is above the threshold are declared duplicates, and pairs below the threshold are declared distinct.\n",
      "\n",
      "To decide where to set the threshold we need to understand what kind of errors result at different levels.\n",
      "If we set the threshold too low, we get more **false positives**, that is, record-pairs we say are duplicates that in reality are not.\n",
      "If we set the threshold too high, we get more **false negatives**, that is, record-pairs that really are duplicates but that we miss.\n",
      "\n",
      "ER algorithms are evaluated by the common metrics of information retrieval and search called **precision** and **recall**.\n",
      "Precision asks of all the record-pairs marked duplicates, what fraction are true duplicates?\n",
      "Recall asks of all the true duplicates in the data, what fraction did we successfully find?\n",
      "As with false positives and false negatives, there is a trade-off between precision and recall.\n",
      "A third metric, called **F-measure**, takes the harmonic mean of precision and recall to measure overall goodness in a single value.:\n",
      "\n",
      "$$Fmeasure = 2 \\frac{precision * recall}{precision + recall}$$\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Exercise 5\n",
      "\n",
      "> **Note**: For this exercise, use the \"gold standard\" mapping from the included file to look up true duplicates, and the results of exercise 4.\n",
      "\n",
      "**a**. Implement functions to count true-positives (true duplicates above the threshold), and false-positives and -negatives.  HINT: To make your functions efficient, you should bin your counts by similarity range."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Look up all similarity scores for true duplicates\n",
      "true_dup_sims = [] # TODO Build this\n",
      "\n",
      "# TODO Just compute true_dup_sim above\n",
      "def truepos(threshold):\n",
      "    return len(true_dup_sims) - falseneg(threshold)\n",
      "\n",
      "# Pre-bin counts of false positives by threshold range\n",
      "nthresholds = 100\n",
      "def bin(similarity):\n",
      "    return int(similarity * nthresholds)\n",
      "\n",
      "fp_counts = {} # TODO Build this.  Should map bin number to count of false-positives\n",
      "\n",
      "# TODO Implement this\n",
      "def falsepos(threshold):\n",
      "    pass\n",
      "\n",
      "# TODO Implement this\n",
      "def falseneg(threshold):\n",
      "    pass\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**b**. What is the relationship between false-positives and -negatives (and true-positives and -negatives) on the one hand and precision and recall on the other?\n",
      "Use the functions from part **a** to implement functions to compute precision, recall, F-measure as a function of threshold value."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# TODO Implement this (returns a float)\n",
      "def precision(threshold):\n",
      "    pass\n",
      "\n",
      "# TODO Implement this (returns a float)\n",
      "def recall(threshold):\n",
      "    pass\n",
      "\n",
      "# TODO Implement this (returns a float)\n",
      "def fmeasure(threshold):\n",
      "    pass"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**c**. Make line plots of precision, recall, and F-measure as a function of threshold value, for thresholds between 0.0 and 1.0.  You can change `nthresholds` (above in part **a**) to change threshold values to plot."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# For the x-axis\n",
      "thresholds = [float(n) / nthresholds for n in range(0, nthresholds)]\n",
      "\n",
      "# TODO Make a plot.  HINT: Use pylab.plot().  Don't forget labels."
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**d**. Using the plot, pick the optimal threshold value and argue for why it is optimal.\n",
      "If false-positives are considered much worse than false-negatives, how does that change your answer?\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**e**.  State-of-the-art tools can get an F-measure of about 60% on this data set.\n",
      "In this assignment we expect to get an F-measure closer to 40%.\n",
      "Look at some examples of errors (both false-positives and -negatives) and think about what went wrong.\n",
      "What are some ways we might improve our simple classifier?\n",
      "Back up your ideas with examples as much as possible."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO Answer questions"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}